Description

Add an API serve for website gpumode, the api will return sub_id and sub_job_status_id immediately after the job is enqueue with background manager, then pull the data periodically for the update.

Adding queue mechnism to accept run jobs from api request /submission/ and runs the backend job.
the gpumode can then pull the db table to consistently check the status of the submission run.

Why not fastapi background task?

https://fastapi.tiangolo.com/tutorial/background-tasks/

The fastapi background task is ok to use when the background job is <=30 secs such as sending emails, notificaiton. For us, it's not ideal since it uses same event loop.

Long term

Ideally we should handle submission jobs as background jobs using RQ or celery, but due to the time limit, this is a compromise solution. Some improvement can be done in next prs:

1. allow queue recoveries from the db data
2. handle shut down more gracefully, (for instance let running job finish, when heroku restarted)

Or we should change the way we interact with the github runner, instead of keeping listening, we should provide a hook to let the workflow trigger the important update includes upload the result

Checklist

Before submitting this PR, ensure the following steps have been completed:

• Run the slash command /verifyruns on your own server.
  • Run the cluster bot on your server:

    python discord-bot.py

  • Start training runs with the slash command /verifyruns.
  • Verify that the bot eventually responds with:

    ✅ All runs completed successfully!
    

    (It may take a few minutes for all runs to finish. In particular, the GitHub
    runs may take a little longer. The Modal run is typically quick.)
    For more information on running a cluster bot on your own server, see
    README.md.